Technologies for timestamping with error correction

ABSTRACT

Technologies for timestamping data packets with forward error correction is disclosed. Alignment markers may be inserted in a data stream in order to assist with synchronization on the receiving end. After insertion of the alignment markers, a start of frame delimiter or other trigger may be detected, triggering a timestamp corresponding to the start of frame delimiter or other trigger. The data and the timestamp are sent to a remote compute device, which may timestamp the data before removing the alignment markers. With this approach, insertion of the alignment markers does not lead to a deviation in the timestamp of the sending compute device or the receiving compute device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 17/503,817, filed Oct. 18, 2021, which is a continuation ofU.S. patent application Ser. No. 15/941,854, filed Mar. 30, 2018, whichclaims priority under 35 U.S.C. § 119(e) to U.S. Provisional PatentApplication No. 62/619,689, filed Jan. 19, 2018. Each of these priorapplications is hereby incorporated herein by reference in its entirety.

BACKGROUND

Timestamping packets that are sent or received over a network isimportant for several applications. Determination of a time of sendingand arrival of a packet may be used to synchronize a time of day counterbetween the sending and receiving compute devices, which may benecessary for time-sensitive applications such as certain wirelesscommunication protocols.

A timestamp of when a packet is sent or received may have severalsources of inaccuracy. For example, a packet may be scrambled, requiringdescrambling before a pattern can be recognized for timestamping, or thepacket may be transferred between domains with asynchronous clocks ascompared to the clock used for timestamping, both of which maycontribute to the inaccuracy in timestamping when a packet actuallyarrives at a device.

BRIEF DESCRIPTION OF THE DRAWINGS

The concepts described herein are illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. Where considered appropriate, referencelabels have been repeated among the figures to indicate corresponding oranalogous elements.

FIG. 1 is a simplified block diagram of at least one embodiment of acompute device for timestamping data packets;

FIG. 2 is a block diagram of at least one embodiment of an environmentthat may be established by the compute device of FIG. 1 ;

FIGS. 3 & 4 is a simplified flow diagram of at least one embodiment of amethod for performing time stamping with error correction for data beingsent by a network interface controller of the compute device of FIG. 1 ;and

FIGS. 5 & 6 is a simplified flow diagram of at least one embodiment of amethod for performing time stamping with error correction for data beingreceived by the network interface controller of the compute device ofFIG. 1 .

DETAILED DESCRIPTION OF THE DRAWINGS

While the concepts of the present disclosure are susceptible to variousmodifications and alternative forms, specific embodiments thereof havebeen shown by way of example in the drawings and will be describedherein in detail. It should be understood, however, that there is nointent to limit the concepts of the present disclosure to the particularforms disclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives consistent with the presentdisclosure and the appended claims.

References in the specification to “one embodiment,” “an embodiment,”“an illustrative embodiment,” etc., indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may or may not necessarily includethat particular feature, structure, or characteristic. Moreover, suchphrases are not necessarily referring to the same embodiment. Further,when a particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to effect such feature, structure,or characteristic in connection with other embodiments whether or notexplicitly described. Additionally, it should be appreciated that itemsincluded in a list in the form of “at least one A, B, and C” can mean(A); (B); (C): (A and B); (B and C); (A and C); or (A, B, and C).Similarly, items listed in the form of “at least one of A, B, or C” canmean (A); (B); (C): (A and B); (B and C); (A and C); or (A, B, and C).

The disclosed embodiments may be implemented, in some cases, inhardware, firmware, software, or any combination thereof. The disclosedembodiments may also be implemented as instructions carried by or storedon one or more transitory or non-transitory machine-readable (e.g.,computer-readable) storage medium, which may be read and executed by oneor more processors. A machine-readable storage medium may be embodied asany storage device, mechanism, or other physical structure for storingor transmitting information in a form readable by a machine (e.g., avolatile or non-volatile memory, a media disc, or other media device).

In the drawings, some structural or method features may be shown inspecific arrangements and/or orderings. However, it should beappreciated that such specific arrangements and/or orderings may not berequired. Rather, in some embodiments, such features may be arranged ina different manner and/or order than shown in the illustrative figures.Additionally, the inclusion of a structural or method feature in aparticular figure is not meant to imply that such feature is required inall embodiments and, in some embodiments, may not be included or may becombined with other features.

Referring now to FIG. 1 , an illustrative compute device 100 isconfigured to perform timestamping of data packets that are either sentor received by a network interface controller 110 of the compute device100. In use, when data is received by the network interface controller110 from another component of the compute device 100, the networkinterface controller 110 performs several steps as part of processingthe data before sending it over the network. For example, in anillustrative embodiment, the data is received on a 32-bit bus using anelastic store phase first in, first out (FIFO) coder. The phase FIFOcoder may periodically insert alignment markers and may delete otherdata such as idle control characters. It should be appreciated that theinsertion of alignment markers and/or deletion of other data such asidle control characters may impact the timing of when the data is sentover the network and could lead to a timestamp with lower precision.After the phase FIFO, the data on the 32-bit bus is transferred to a64-bit bus. Then, each block of 64 bits is encoded and scrambled to 66bits. As part of encoding the 64 bits into 66 bits, the networkinterface controller 110 monitors for a trigger pattern, such as a startof frame delimiter of an Ethernet packet, and triggers a timestamp whenthe start of frame delimiter is detected. It should be appreciated that,since the timestamp is done after the alignment markers are insertedand/or other data such as idle control characters are deleted, thetimestamp will not be impacted due to that insertion or deletion.Subsequently, a set of four blocks of 66 bits each (264 bits total) istranscoded to a block of 257 bits. Several blocks of 257 bits are thenencoded with a forward error correction algorithm, such as Reed-Solomon.The encoded blocks are then sent out to the network, such as on anelectrical cable or optical cable.

The compute device 100 may be embodied as any type of compute devicecapable of managing data packets and performing the functions describedherein. For example, the compute device 100 may be embodied as orotherwise be included in, without limitation, a server computer, anembedded computing system, a System-on-a-Chip (SoC), a multiprocessorsystem, a processor-based system, a consumer electronic device, asmartphone, a cellular phone, a desktop computer, a tablet computer, anotebook computer, a laptop computer, a network device, a router, aswitch, a networked computer, a wearable computer, a handset, amessaging device, a camera device, and/or any other computing device.The illustrative compute device 100 includes the processor 102, a memory104, an input/output (I/O) subsystem 106, a system clock 108, a networkinterface controller 110, and data storage 112. In some embodiments, oneor more of the illustrative components of the compute device 100 may beincorporated in, or otherwise form a portion of, another component. Forexample, the memory 104, or portions thereof, may be incorporated in theprocessor 102 in some embodiments.

The processor 102 may be embodied as any type of processor capable ofperforming the functions described herein. For example, the processor102 may be embodied as a single or multi-core processor(s), a single ormulti-socket processor, a digital signal processor, a graphicsprocessor, a microcontroller, or other processor orprocessing/controlling circuit. Similarly, the memory 104 may beembodied as any type of volatile or non-volatile memory or data storagecapable of performing the functions described herein. In operation, thememory 104 may store various data and software used during operation ofthe compute device 100 such as operating systems, applications,programs, libraries, and drivers. The memory 104 is communicativelycoupled to the processor 102 via the I/O subsystem 106, which may beembodied as circuitry and/or components to facilitate input/outputoperations with the processor 102, the memory 104, and other componentsof the compute device 100. For example, the I/O subsystem 106 may beembodied as, or otherwise include, memory controller hubs, input/outputcontrol hubs, firmware devices, communication links (i.e.,point-to-point links, bus links, wires, cables, light guides, printedcircuit board traces, etc.) and/or other components and subsystems tofacilitate the input/output operations. In some embodiments, the I/Osubsystem 106 may form a portion of a system-on-a-chip (SoC) and beincorporated, along with the processor 102, the memory 104, and othercomponents of the compute device 100 on a single integrated circuitchip.

The system clock 108 may be embodied as any type of device, circuit,and/or collection of devices or circuits capable of generating a clocksignal usable by other components of the compute device 100. Forexample, the system clock 108 may be embodied as, or otherwise include,a crystal oscillator-based circuit. The clock signal may be used byvarious components of the compute device 100 to synchronize timing ofcommunication. For example, the processor 102 and the memory 104 mayeach be synchronized to the system clock 108. The clock signal may beany type of clock signal, such as a square wave or a sine wave. Thefrequency of the clock signal generated by the system clock 108 may beany suitable frequency, such as 100, 200, or 500 megahertz (MHz).

The network interface controller 110 may be embodied as any type ofinterface capable of interfacing the compute device 100 with othercompute devices, such as over a network. In some embodiments, thenetwork interface controller 110 may be referred to as a host fabricinterface (HFI). The network interface controller 110 may be capable ofinterfacing with any appropriate cable type, such as an electrical cableor an optical cable, and/or may be capable of interfacing with awireless signal, such as through one or more antennae. The networkinterface controller 110 may be configured to use any one or morecommunication technology and associated protocols (e.g., Ethernet,Bluetooth®, Wi-Fi®, WiMAX, near field communication (NFC), etc.). Thenetwork interface controller 110 may be located on silicon separate fromthe processor 102, or the network interface controller 110 may beincluded in a multi-chip package with the processor 102, or even on thesame die as the processor 102. The network interface controller 110 maybe embodied as one or more add-in-boards, daughtercards, networkinterface cards, controller chips, chipsets, specialized components suchas a field programmable gate array (FPGA) or application specificintegrated circuit (ASIC), or other devices that may be used by thecompute device 100 to connect with another compute device. In someembodiments, network interface controller 110 may be embodied as part ofa system-on-a-chip (SoC) that includes one or more processors, orincluded on a multichip package that also contains one or moreprocessors. In some embodiments, the network interface controller 110may include a local processor (not shown) and/or a local memory (notshown) that are both local to the network interface controller 110. Insuch embodiments, the local processor of the network interfacecontroller 110 may be capable of performing one or more of the functionsof the processor 102 described herein. Additionally or alternatively, insuch embodiments, the local memory of the network interface controller110 may be integrated into one or more components of the compute device100 at the board level, socket level, chip level, and/or other levels.

The illustrative network interface controller 110 includes one or morenetwork interface controller clocks (NIC clocks) 114. In theillustrative embodiment, one of the NIC clocks 114 is used fortransmitting a signal to another compute device. The NIC clock 114 usedfor transmission may be similar to the system clock 108, the descriptionof which will not be repeated in the interest of clarity. In theillustrative embodiment, another one of the NIC clocks 114 is used forreceiving a signal from another compute device. The NIC clock 114 usedfor receiving may operate in a different manner from the system clock108, such as by recovering a clock from the signal received by thenetwork interface controller 110. It should be appreciated that the NICclocks 114 may operate at a different frequency than the system clock108, and the NIC clocks 114 and the system clock 108 may be free-running(i.e., operate independently of each other with no fixed timingrelationship between them). It should further be appreciated thatdifferent components of the network interface controller 110 may operateon different clocks, including the NIC clock(s) 114 and the system clock108.

The data storage 112 may be embodied as any type of device or devicesconfigured for the short-term or long-term storage of data. For example,the data storage 112 may include any one or more memory devices andcircuits, memory cards, hard disk drives, solid-state drives, or otherdata storage devices.

In some embodiments, the compute device 100 may include other oradditional components, such as those commonly found in a compute device.For example, the compute device 100 may also have a display 116 and/orperipheral devices 118. The peripheral devices 118 may include akeyboard, a mouse, etc. The display 116 may be embodied as any type ofdisplay on which information may be displayed to a user of the computedevice 100, such as a touchscreen display, a liquid crystal display(LCD), a light emitting diode (LED) display, a cathode ray tube (CRT)display, a plasma display, an image projector (e.g., 2D or 3D), a laserprojector, a heads-up display, and/or other display technology.

Referring now to FIG. 2 , in an illustrative embodiment, the computedevice 100 establishes an environment 200 during operation. Theillustrative environment 200 includes a network interface manager 202.The various modules of the environment 200 may be embodied as hardware,software, firmware, or a combination thereof. For example, the variousmodules, logic, and other components of the environment 200 may form aportion of, or otherwise be established by, the processor 102 or otherhardware components of the compute device 100 such as the networkinterface controller 110. As such, in some embodiments, one or more ofthe modules of the environment 200 may be embodied as circuitry orcollection of electrical devices (e.g., a network interface managercircuit 202, etc.). It should be appreciated that, in such embodiments,one or more of the circuits (e.g., the network interface manager circuit202, etc.) may form a portion of one or more of the processor 102, thememory 104, the I/O subsystem 106, the network interface controller 110,the data storage 112, and/or other components of the compute device 100.Additionally, in some embodiments, one or more of the illustrativemodules may form a portion of another module and/or one or more of theillustrative modules may be independent of one another. Further, in someembodiments, one or more of the modules of the environment 200 may beembodied as virtualized hardware components or emulated architecture,which may be established and maintained by the processor 102 or othercomponents of the compute device 100. It should be appreciated that someof the functionality of one or more of the modules of the environment200 may require a hardware implementation, in which case embodiments ofmodules which implement such functionality will be embodied at leastpartially as hardware.

The network interface manager 202, which may be embodied as hardware,firmware, software, virtualized hardware, emulated architecture, and/ora combination thereof as discussed above, is configured to control thenetwork interface controller 110, including the interaction of thenetwork interface controller 110 with the physical medium to which it isattached and the processing of incoming and outgoing signals which thenetwork interface controller 110 receives and sends. The networkinterface manager 202 includes a physical medium attachment sublayer204, one or more gearboxes 206, a physical coding sublayer 208, aforward error correction (FEC) encoder/decoder 210, and a data packettimestamper 212.

The physical medium attachment sublayer 204 is configured to sample anincoming signal at the physical medium to which it is attached andprovide a signal usable for other components of the network interfacecontroller 110, such as a serial electrical signal at standardizedvoltage levels. The physical medium attachment sublayer 204 may operateat any suitable rate or bandwidth, such 10 gigabits per second (Gbps),25 Gbps, 40 Gbps, 100 Gbps, or higher or lower than those rates. Thephysical medium attachment sublayer 204 is also configured to sendoutgoing signals received from other components of the network interfacecontroller 110 to the physical medium to which it is attached. In theillustrative embodiment, the physical medium attachment sublayer 204synchronizes a NIC clock 114 to the incoming signal, which is used bythe physical medium attachment sublayer 204 and other components of thenetwork interface controller 110 for processing the incoming signal. Theillustrative physical medium attachment sublayer 204 also uses a NICclock 114 that is free-running relative to other clocks for sending asignal over the physical medium and for processing the signal before itis sent over the physical medium.

The illustrative physical medium attachment includes aserializer/deserializer (SerDes) 214. The SerDes 214 is configured totransfer a serial signal to a parallel signal and to transfer a parallelsignal to a serial signal. In the illustrative embodiment, the physicalmedium is sampled or driven serially, and the SerDes 214 deserializesincoming signals from the physical medium to a bus for use by thephysical medium attachment sublayer 204 and serializes signals from thephysical medium attachment sublayer 204 to be sent on the physicalmedium. The SerDes 212 may transfer between a serial interface andparallel interface of any suitable bus size, such as 16 or 40 bits. Insome embodiments, the signal physical medium may not be sent out as asingle serial signal, but may be sent out as a parallel signal or asseveral separate serial signals.

The gearbox 206 is configured to transfer a parallel bus of a firstwidth to/from a parallel bus of a second width. For example, the gearbox206 may be used to transfer between a parallel bus of 257 bits at theFEC encoder/decoder 210 to/from a parallel bus of 40 bits at thephysical medium attachment sublayer 204.

The physical coding sublayer 208 is configured to perform coding on theincoming and outgoing signals. In the illustrative embodiment, thephysical coding sublayer 208 performs a 64b/66b encoding to decode 66bits to 64 bits or to encode 64 bits to 66 bits using the 64b/66bencoding algorithm described in IEEE Std 802.3-2015, approved Sep. 3,2015, by the IEEE-SA Standards Board. To do so, in the illustrativeembodiment, the physical coding layer 208 takes 64 bits of outgoingdata, adds 2 bits as a synchronization header, and then scrambles the 64bits. For incoming data, the illustrative physical coding sublayer 208takes 66 bits of data, removes the first two bits of the synchronizationheader, and descrambles the remaining two bits. The two bits in theillustrative synchronization header are only valid if they are “01” or“10,” and they perform two functions in the illustrative embodiment.First, they act as an indication of the start of 66 bit blocks becausethere should always be a transition from zero to one or one to zeroevery 66 bits. Second, they indicate whether the bits in the following64-bit block are control bits or data bits. In other embodiments, thephysical coding sublayer 208 may perform additional or alternativeencoding/decoding, such as 4 b/5 b encoding/decoding or 8 b/10 bencoding/decoding. It should be appreciated that, in the illustrativeembodiment, the incoming signal may not be able to be processed untilafter the physical coding sublayer 208 decodes the signal. For example,the trigger pattern that is being detected as a trigger for timestampingmay only be able to be detected after decoding is complete.

The illustrative physical coding sublayer 208 may delete data such asidle control characters from outgoing data. In the illustrativeembodiment, idle control characters may be deleted from an interpacketgap period. The physical coding sublayer 208 may periodically insert oneor more alignment markers, such as after every 16,383 64-bit blocks orafter every 81,916 64-bit blocks. In some embodiments, alignment markersmay be inserted in each of several data lanes. For incoming data, withmultiple data lanes, the illustrative physical coding sublayer 208 mayuse the alignment markers to synchronize the data between differentlanes. Additionally or alternatively, in some embodiments, the alignmentmarkers may be used as codeword markers. As used herein, an alignmentmarker refers to a marker that may be used to align multiple data lanesor to a marker that may be used to identify codewords. The illustrativephysical coding sublayer 208 removes the alignment markers (regardlessof whether there is one lane or more than one lane) and inserts idlecontrol characters in the interpacket gap period. It should beappreciated that the insertion of alignment markers and/or deletion ofother data such as idle control characters may impact the timing of whenthe data is sent over the network and could lead to a timestamp withlower precision if not corrected for. In some embodiments, the physicalcoding sublayer 208 may receive data with use of an elastic store/phasefirst-in, first-out buffer. Use of such a buffer may allow the physicalcoding sublayer 208 to insert alignment markers and delete data such asidle control characters in a way that is less disruptive to the dataflow through the components of the NIC 110.

The FEC encoder/decoder 210 is configured to perform forward errorcorrection by encoding an outgoing bitstream and decoding an incomingbitstream. As part of the encoding, the FEC encoder/decoder 210 maytranscode four 66 bit blocks of outgoing data to one 257 bit block. Inthe illustrative embodiment, the FEC encoder/decoder 210 may removealignment markers, remap them, and insert them into the bitstream ascodeword markers without encoding. The remapped alignment markers mayhave bits shifted or changed as compared to the alignment markers beforeremapping. In the illustrative embodiment, 4 alignment markers of 66bits each are remapped to 256 bits plus a 1-bit pad for a total of 257bits. In the illustrative embodiment, the alignment markers can be usedto identify the starting position of codewords in the FEC algorithm,which can be used to decode the encoded data. For that reason, thealignment markers also function as codeword markers. It should beappreciated that, as part of the decoding process, the FECencoder/decoder 210 may reinsert the alignment markers to the sameposition as they were removed, which would not cause any variable delayin the timestamping based on detection of a trigger pattern. In theillustrative embodiment, the FEC encode/decoder uses a Reed-Solomonerror correcting algorithm with 10-bit symbols by encoding 20 257-bitblocks (i.e., 5,140 bits) with 140 parity bits. The resulting 5,280-bitsform a codeword of the FEC algorithm. It should be appreciated that theremapped alignment markers may also serve as codeword markers,indicating the boundaries between codewords. In some embodiments,different approaches to the FEC encoding may be done, such as by usingdifferent encoding algorithms, different numbers of blocks, or differentnumbers of parity bits may be used.

The FEC encoder/decoder 210 is configured to decode incoming data in asimilar manner as it encodes outgoing data. The FEC encoder/decoder 210may lock onto alignment markers (which also act as codeword markers) tomark the boundaries between codewords. Each codeword is decoded, thenthe alignment markers are removed. The 257-bit blocks are transcoded to66-bit blocks in a complementary manner to the process described above.Likewise, the alignment markers are remapped to 66-bit blocks in acomplementary manner to the process described above. The alignmentmarkers are reinserted to the data in the same location that they wereremoved.

The trigger pattern timestamper 212 is configured to trigger a timestampwhen a trigger pattern is being sent or received. The trigger patterntimestamper 212 includes a trigger pattern detector 216 and a triggerpattern reporter 218. The trigger pattern detector 216 is configured todetect a particular trigger patterns such as the start of a data packetor frame in the incoming or outgoing signal. Upon detection of a triggerpattern by the trigger pattern detector 216, the trigger patternreporter 218 is configured to report the presence of a particulartrigger pattern in the incoming or outgoing data by sending acorresponding trigger to the trigger pattern timestamper 212. Thetrigger pattern reporter 218 is configured to send the trigger at afixed time relative to the detection of the relevant trigger pattern sothat no variable latency is introduced by the trigger pattern detector216 or the trigger pattern reporter 218. In the illustrative embodiment,the trigger pattern that the trigger pattern detector 216 and thetrigger pattern reporter 218 are configured to trigger on is a start offrame delimiter in an Ethernet frame. In other embodiments, the triggerpattern detector 216 and/or the trigger pattern reporter 218 may beconfigured to trigger on different trigger patterns, such as a start ofa packet or frame from various other protocols, the end of a packet orframe of various protocols, etc. In the illustrative embodiment, thetrigger pattern timestamper 212 checks for the presence of a triggerpattern in outgoing data including the alignment markers that will bepresent in the outgoing data, such as right before the 64b/66b encodingis performed, and checks for the presence of a trigger pattern inincoming data including the received alignment markers, such as rightafter 64b/66b decoding is performed. In the illustrative embodiment, thealignment markers may be removed, remapped, and reinserted beforelooking for the presence of a trigger pattern, which is acceptablebecause the remapping does not introduce a variable latency in theposition of the trigger pattern.

Referring now to FIG. 3 , in use, the compute device 100 may execute amethod 300 for performing time stamping with error correction is shown.In block 302, the network interface controller (NIC) 110 receives datafrom another component of the compute device 100, such as from theprocessor 102, the memory 104, the I/O subsystem 106, etc.

In block 304, the NIC 110 inserts alignment markers into the data to besent. In the illustrative embodiment, the alignment markers may beinserted at regular intervals, such as after every 16,383 64-bit block.In some embodiments with multiple lanes, the NIC 110 may insertalignment markers in each of the multiple lanes in block 306. The NIC110 may also delete other data such as idle control characters from aninterpacket gap period in block 308. It should be appreciated that theidle control characters may be deleted to compensate for the insertionof the alignment markers, but the alignment markers do not necessarilyappear at the same location as the idle control characters. In someembodiments, the NIC 110 may insert the alignment markers and deletedata such as idle control characters as data is passed through anelastic store/phase first-in, first-out buffer. Use of such a buffer mayallow the NIC 110 to insert alignment markers and delete data such asidle control characters in a way that is less disruptive to the dataflow through the components of the NIC 110.

In block 310, the NIC 110 checks for the presence of a trigger patternin the data. In the illustrative embodiment, the trigger pattern is astart of frame delimiter in an Ethernet frame. In other embodiments, thetrigger pattern may be different, such as a start of a packet or framefrom various other protocols, the end of a packet or frame of variousprotocols, etc. If the NIC 110 detects a trigger pattern, the method 300proceeds to block 312, in which the NIC 110 triggers a timestamp basedon the detection of the trigger pattern. The NIC 110 may record thetimestamp corresponding to the trigger pattern found and may use thetimestamp to perform various functions, such as a time synchronizationwith a remote compute device.

Referring back to block 310, if the NIC 110 does not detect a triggerpattern in the data, the method 300 jumps to block 314, in which the NIC110 encodes the data. In the illustrative embodiment, the NIC 110encodes the data by encoding 64 bits to 66 bits using the 64b/66bencoding algorithm described in IEEE Std 802.3-2015, approved Sep. 3,2015, by the IEEE-SA Standards Board. To do so, in the illustrativeembodiment, the NIC 110 adds 2 bits as a synchronization header beforethe 64-bit block in block 314. The NIC 110 then scrambles the 64bit-block and prepends the two synchronization bits to the scrambled 64bits in block 318. It should be appreciated that, in the illustrativeembodiment, the NIC 110 does not scramble the alignment markers,although the NIC 110 may still prepend two the synchronization headerbits to create a 66-bit alignment marker from a 64-bit alignment marker.The two bits in the illustrative synchronization header are only validif they are “01” or “10.” The synchronization header bits indicatewhether the bits in the following 64-bit block are control bits or databits. In other embodiments, the physical coding sublayer 208 may performadditional or alternative encoding/decoding, such as 4 b/5 bencoding/decoding or 8 b/10 b encoding/decoding. It should beappreciated that, in the illustrative embodiment, the outgoing data maynot be able to be processed after the data is scrambled. For example, inthe illustrative embodiment, the trigger pattern may not able to bedetected when scrambled. However, certain other operations may takeplace before checking for the presence of the trigger pattern, such asadding the 2-bit synchronization header.

The method 300 proceeds to block 320 in FIG. 4 , in which the NIC 110removes the alignment markers in block 320. The NIC 110 then remaps thealignment markers in block 322. The remapped alignment markers may havebits shifted or changed as compared to the alignment markers beforeremapping. In the illustrative embodiment, 4 alignment markers of 66bits each are remapped to 256 bits plus a 1-bit pad for a total of 257bits.

The NIC 110 then transcodes the 66-bit blocks of synchronization headersand data backs by transcoding 4 66-bit blocks to one 257-bit block inblock 324. The NIC 110 then reinserts the remapped alignment markersinto the data in block 326. In the illustrative embodiment, thealignment markers can be used to identify the starting position ofcodewords in the FEC algorithm, which can be used to decode the encodeddata. For that reason, the alignment markers also function as codewordmarkers. It should be appreciated that the NIC 110 may reinsert thealignment markers to the same position as they were removed, which wouldnot cause any variable delay in the timestamping based on detection of atrigger pattern.

In block 328, the NIC 110 performs forward error correction encoding onthe data. In the illustrative embodiment, the FEC encode/decoder uses aReed-Solomon error correcting algorithm with 10-bit symbols by encoding20 257-bit blocks (i.e., 5,140 bits) with 140 parity bits. The resulting5,280-bits form a codeword of the FEC algorithm. In some embodiments,different approaches to the FEC encoding may be done, such as by usingdifferent encoding algorithms, different numbers of blocks, or differentnumbers of parity bits may be used.

In block 330, the NIC 110 sends the encoded data over a network. In someembodiments, the NIC 110 may send the encoded data over a physicalmedium directly to a destination compute device or the data mayotherwise not be sent to a network of compute devices.

Referring now to FIG. 5 , in use, a compute device 100 may execute amethod 500 for performing time stamping with error correction is shown.The method 500 is used to timestamp data coming in to the NIC 110 from anetwork or from another compute device. The method 500 is complementaryto the method 300, which is a method used to timestamp data going outfrom the NIC 110 to a network or to another compute device. Because manyof the details are the same between the two methods, some of the detailwill be omitted in the description of the method 500 in the interest ofclarity.

In block 502, the NIC 110 received data from a network. In someembodiments, the NIC 110 may receive the data over a physical mediumdirectly from a source compute device or the data may otherwise not bereceived from a network of compute devices.

In block 504, the NIC 110 performs forward error correction decoding. Aspart of performing error correction decoding, the NIC 110 may lock ontoalignment markers (which may function as codeword markers) in order todetermine the beginning of codewords to decode. The forward errorcorrection decoding may be complementary to the encoding done in block328 of the method 300.

In block 506, the NIC 110 removes the alignment markers from the data.In block 508, the NIC 110 remaps the alignment markers. The remappedalignment markers may have bits shifted or changed as compared to thealignment markers before remapping. In the illustrative embodiment, 4alignment markers in a block of 257 bits are remapped to 4 66-bitalignment markers.

In block 510, the NIC 110 transcodes the decoded data blocks. In theillustrative embodiment, the NIC 110 transcodes a block of 257 bits tofour 66-bit blocks. In block 512, the remapped alignment marker isreinserted into the data. It should be appreciated that the NIC 110 mayreinsert the alignment markers to the same position as they wereremoved, which would not cause any variable delay in the timestampingbased on detection of a trigger pattern.

The method 500 proceeds to block 514 in FIG. 6 , in which the NIC 110performs another level of decoding. In the illustrative embodiment, theNIC 110 performs a 66b/64b decoding. The 66b/64b decoding may becomplementary to the 66b/64b encoding performed in block 314 of themethod 300. The NIC 110 removes a synchronization header from each66-bit block in block 516. In block 518, the NIC 110 descrambles each64-bit blocks except the 64-bit blocks corresponding to the alignmentmarkers.

In block 520, the NIC 110 checks for the presence of a trigger patternin the data. In the illustrative embodiment, the trigger pattern is astart of frame delimiter in an Ethernet frame. In other embodiments, thetrigger pattern may be different, such as a start of a packet or framefrom various other protocols, the end of a packet or frame of variousprotocols, etc. If the NIC 110 detects a trigger pattern, the method 500proceeds to block 522, in which the NIC 110 triggers a timestamp basedon the detection of the trigger pattern. The NIC 110 may record thetimestamp corresponding to the trigger pattern found and may use thetimestamp to perform various functions, such as a time synchronizationwith a remote compute device.

Referring back to block 520, if the NIC 110 does not detect a triggerpattern in the data, the method 500 jumps to block 524, in which the NIC110 removes alignment markers from the data. In some embodiments withmultiple lanes, the NIC 110 may remove alignment markers in each of themultiple lanes in block 526. The NIC 110 may also insert other data suchas idle control characters to an interpacket gap period in block 528. Itshould be appreciated that the idle control characters may be insertedto compensate for the deletion of the alignment markers, but thealignment markers do not necessarily appear at the same location as theidle control characters. In some embodiments, the NIC 110 may delete thealignment markers and insert other data such as idle control charactersas data is passed through an elastic store/phase first-in, first-outbuffer. Use of such a buffer may allow the NIC 110 to delete alignmentmarkers and insert other data such as idle control characters in a waythat is less disruptive to the data flow through the components of theNIC 110. In block 520, the NIC 110 passes the data to another componentof the compute device 100, such as from the processor 102, the memory104, the I/O subsystem 106, etc.

It should be appreciated that, in some embodiments, the compute device100 may perform accurate timestamping while also employing errorcorrection in a different manner. For example, in some embodiments, thephysical coding sublayer 208 may not insert alignment markers and maynot delete idle characters or other data to compensate for the alignmentmarkers. Rather, in such embodiments, the FEC encoder/decoder 210 mayinsert alignment (or codeword) markers into outgoing data and deleteidler characters and/or other data to compensate for insertion of thealignment markers. To do so, the FEC encoder/decoder 210 may decode theoutgoing data using a 66b/64b decoding in order to recognize idlecontrol characters. After this decoding, the FEC encoder/decoder 210 mayinsert the alignment markers. After the insertion of the alignmentmarkers and before re-encoding the outgoing data with a 66b/64bencoding, the FEC encoder/decoder 210 may determine whether the outgoingdata includes a trigger pattern, such as a start of frame delimiter, ina similar manner as the physical coding sublayer 208 above. The FECencoder/decoder 210 may then trigger a timestamp based on the detectionof the trigger feature. It should be appreciated that detection of thetrigger feature in the FEC encoder/decoder 210 after insertion of thealignment markers by the FEC encoder/decoder 210 in such embodiments hasthe same advantages of avoiding the variable latency from the insertionof the alignment markers as the embodiments described above in regard toFIGS. 3-6 . For outgoing data, the FEC encoder/decoder 210 may performcomplementary actions to those described above in a similar way that theactions described in FIGS. 5 & 6 are complementary to the actionsdescribed in FIGS. 3 & 4 .

EXAMPLES

Illustrative examples of the devices, systems, and methods disclosedherein are provided below. An embodiment of the devices, systems, andmethods may include any one or more, and any combination of, theexamples described below.

Example 1 includes a compute device for timestamping data packets, thecompute device comprising a processor; a memory; and a network interfacecontroller to receive data from another component of the compute device;insert one or more alignment markers in the data; detect a triggerpattern in the data after insertion of the one or more alignmentmarkers; trigger a timestamp corresponding to detection of the triggerpattern; and send the data to a remote compute device.

Example 2 includes the subject matter of Example 1, and wherein thenetwork interface controller is further to remove the one or morealignment markers from the data; remap the one or more alignmentmarkers; encode the data with a forward error correction algorithm; andinsert the one or more alignment markers as codeword markers in thedata.

Example 3 includes the subject matter of any of Examples 1 and 2, andwherein the trigger pattern is a start of frame delimiter.

Example 4 includes the subject matter of any of Examples 1-3, andwherein the network interface controller is further to addsynchronization headers to each of a plurality of 64-bit blocks; andscramble each of the plurality of 64-bit blocks, wherein to detect thetrigger pattern in the data comprises to detect the trigger pattern inthe plurality of 64-bit blocks after addition of the synchronizationheaders and before scrambling of the plurality of 64-bit blocks.

Example 5 includes the subject matter of any of Examples 1-4, andwherein the data comprises a plurality of 64-bit blocks, wherein thenetwork interface controller is further to scramble each of theplurality of 64-bit blocks after insertion of the alignment markerswithout scrambling the alignment markers.

Example 6 includes the subject matter of any of Examples 1-5, andwherein the network interface controller is further to remove a portionof an interpacket gap from the data to compensate for the insertion ofthe alignment markers.

Example 7 includes the subject matter of any of Examples 1-6, andwherein the network interface controller is further to receiveadditional data from the remote compute device; detect the triggerpattern in the additional data; trigger a timestamp corresponding todetection of the trigger pattern in the additional data; and remove oneor more alignment markers in the additional data after detection of thetrigger pattern in the additional data.

Example 8 includes the subject matter of any of Examples 1-7, andwherein to send the data to the remote compute device comprises to sendthe data to the remote compute device at a rate of at least 25 gigabitsper second.

Example 9 includes a method for timestamping data packets by a networkinterface controller of a compute device, the method comprisingreceiving, by the network interface controller, data from anothercomponent of the compute device; inserting, by the network interfacecontroller, one or more alignment markers in the data; detecting, by thenetwork interface controller, a trigger pattern in the data afterinsertion of the one or more alignment markers; triggering, by thenetwork interface controller, a timestamp corresponding to detection ofthe trigger pattern; and sending, by the network interface controller,the data to a remote compute device.

Example 10 includes the subject matter of Example 9, and furtherincluding removing, by the network interface controller, the one or morealignment markers from the data; remapping, by the network interfacecontroller, the one or more alignment markers; encoding, by the networkinterface controller, the data with a forward error correctionalgorithm; and inserting, by the network interface controller, the oneor more alignment markers as codeword markers in the data.

Example 11 includes the subject matter of any of Examples 9 and 10, andwherein the trigger pattern is a start of frame delimiter.

Example 12 includes the subject matter of any of Examples 9-11, andfurther including adding, by the network interface controller,synchronization headers to each of a plurality of 64-bit blocks; andscrambling, by the network interface controller, each of the pluralityof 64-bit blocks, wherein detecting the trigger pattern in the datacomprises detecting the trigger pattern in the plurality of 64-bitblocks after addition of the synchronization headers and beforescrambling of the plurality of 64-bit blocks.

Example 13 includes the subject matter of any of Examples 9-12, andwherein the data comprises a plurality of 64-bit blocks, furthercomprising scrambling, by the network interface controller, each of theplurality of 64-bit blocks after insertion of the alignment markerswithout scrambling the alignment markers.

Example 14 includes the subject matter of any of Examples 9-13, andfurther including removing, by the network interface controller, aportion of an interpacket gap from the data to compensate for theinsertion of the alignment markers.

Example 15 includes the subject matter of any of Examples 9-14, andfurther including receiving, by the network interface controller,additional data from the remote compute device; detecting, by thenetwork interface controller, the trigger pattern in the additionaldata; triggering, by the network interface controller, a timestampcorresponding to detection of the trigger pattern in the additionaldata; and removing, by the network interface controller, one or morealignment markers in the additional data after detection of the triggerpattern in the additional data.

Example 16 includes one or more machine-readable media comprising aplurality of instructions stored thereon that, when executed, causes acompute device to receive data from another component of the computedevice; insert one or more alignment markers in the data; detect atrigger pattern in the data after insertion of the one or more alignmentmarkers; trigger a timestamp corresponding to detection of the triggerpattern; and send the data to a remote compute device.

Example 17 includes the subject matter of Example 16, and wherein theplurality of instructions further causes the compute device to removethe one or more alignment markers from the data; remap the one or morealignment markers; encode the data with a forward error correctionalgorithm; and insert the one or more alignment markers as codewordmarkers in the data.

Example 18 includes the subject matter of any of Examples 16 and 17, andwherein the trigger pattern is a start of frame delimiter.

Example 19 includes the subject matter of any of Examples 16-18, andwherein the plurality of instructions further causes the compute deviceto add synchronization headers to each of a plurality of 64-bit blocks;and scramble each of the plurality of 64-bit blocks, wherein to detectthe trigger pattern in the data comprises to detect the trigger patternin the plurality of 64-bit blocks after addition of the synchronizationheaders and before scrambling of the plurality of 64-bit blocks.

Example 20 includes the subject matter of any of Examples 16-19, andwherein the data comprises a plurality of 64-bit blocks, wherein theplurality of instructions further causes the compute device to scrambleeach of the plurality of 64-bit blocks after insertion of the alignmentmarkers without scrambling the alignment markers.

Example 21 includes the subject matter of any of Examples 16-20, andwherein the plurality of instructions further causes the compute deviceto remove a portion of an interpacket gap from the data to compensatefor the insertion of the alignment markers.

Example 22 includes the subject matter of any of Examples 16-21, andwherein the plurality of instructions further causes the compute deviceto receive additional data from the remote compute device; detect thetrigger pattern in the additional data; trigger a timestampcorresponding to detection of the trigger pattern in the additionaldata; and remove one or more alignment markers in the additional dataafter detection of the trigger pattern in the additional data.

Example 23 includes a compute device for timestamping data packets, thecompute device comprising circuitry for receiving, by the networkinterface controller, data from another component of the compute device;means for inserting, by the network interface controller, one or morealignment markers in the data; means for detecting, by the networkinterface controller, a trigger pattern in the data after insertion ofthe one or more alignment markers; means for triggering, by the networkinterface controller, a timestamp corresponding to detection of thetrigger pattern; and circuitry for sending, by the network interfacecontroller, the data to a remote compute device.

Example 24 includes the subject matter of Example 23, and furtherincluding means for removing, by the network interface controller, theone or more alignment markers from the data; means for remapping, by thenetwork interface controller, the one or more alignment markers; meansfor encoding, by the network interface controller, the data with aforward error correction algorithm; and means for inserting, by thenetwork interface controller, the one or more alignment markers ascodeword markers in the data.

Example 25 includes the subject matter of any of Examples 23 and 24, andfurther including means for receiving, by the network interfacecontroller, additional data from the remote compute device; means fordetecting, by the network interface controller, the trigger pattern inthe additional data; means for triggering, by the network interfacecontroller, a timestamp corresponding to detection of the triggerpattern in the additional data; and means for removing, by the networkinterface controller, one or more alignment markers in the additionaldata after detection of the trigger pattern in the additional data.

The invention claimed is:
 1. At least one non-transitorymachine-readable storage medium storing instructions for being executedby at least one machine, the at least one machine being associated withnetwork interface controller circuitry, the network interface controllercircuitry being for use with a physical medium, the physical mediumbeing for use in Ethernet protocol frame transmission between thenetwork interface controller circuitry and a remote device, theinstructions when executed by the at least one machine resulting in thenetwork interface controller circuitry being configured for performanceof operations comprising: receiving, via the physical medium, Ethernetprotocol frame data, the Ethernet protocol frame data comprising atleast one alignment marker and data associated with triggering of atimestamp; detecting the data associated with the triggering of thetimestamp; triggering, based upon the data associated with thetriggering of the timestamp, the timestamp; and at a physical layer,after the triggering of the timestamp, (1) removing the at least onealignment marker from the Ethernet protocol frame data, and (2)inserting at least one idle character into the Ethernet protocol framedata.
 2. The at least one non-transitory machine-readable storage mediumof claim 1, wherein: the data associated with the triggering of thetimestamp comprises start of frame delimiter data.
 3. The at least onenon-transitory machine-readable storage medium of claim 2, wherein: theat least one alignment marker comprises codeword marker data.
 4. The atleast one non-transitory machine-readable storage medium of claim 3,wherein: the operations also comprise deleting, from the Ethernetprotocol frame data, idle control character data.
 5. The at least onenon-transitory machine-readable storage medium of claim 4, wherein: thenetwork interface controller circuitry comprises physical mediumattachment circuitry and physical coding sublayer circuitry for use inassociation with processing the Ethernet protocol frame data.
 6. Amethod implemented using network interface controller circuitry, thenetwork interface controller circuitry being for use with a physicalmedium, the physical medium being for use in Ethernet protocol frametransmission between the network interface controller circuitry and aremote device, the method comprising: receiving, via the physicalmedium, Ethernet protocol frame data, the Ethernet protocol frame datacomprising at least one alignment marker and data associated withtriggering of a timestamp; detecting the data associated with thetriggering of the timestamp; triggering, based upon the data associatedwith the triggering of the timestamp, the timestamp; and at a physicallayer, after the triggering of the timestamp, (1) removing the at leastone alignment marker from the Ethernet protocol frame data, and (2)inserting at least one idle character into the Ethernet protocol framedata.
 7. The method of claim 6, wherein: the data associated with thetriggering of the timestamp comprises start of frame delimiter data. 8.The method of claim 7, wherein: the at least one alignment markercomprises codeword marker data.
 9. The method of claim 8, wherein: themethod also comprises deleting, from the Ethernet protocol frame data,idle control character data.
 10. The method of claim 9, wherein: thenetwork interface controller circuitry comprises physical mediumattachment circuitry and physical coding sublayer circuitry for use inassociation with processing the Ethernet protocol frame data. 11.Network interface controller circuitry for use with a physical medium,the physical medium being for use in Ethernet protocol frametransmission between the network interface controller circuitry and aremote device, the network interface controller circuitry comprising:circuitry to receive, via the physical medium, Ethernet protocol framedata, the Ethernet protocol frame data comprising at least one alignmentmarker and data associated with triggering of a timestamp; circuitry todetect the data associated with the triggering of the timestamp;circuitry to trigger, based upon the data associated with the triggeringof the timestamp, the timestamp; and physical layer circuitry to (1)remove, after the triggering of the timestamp, the at least onealignment marker from the Ethernet protocol frame data, and (2) toinsert, after the triggering of the timestamp, at least one idlecharacter into the Ethernet protocol frame data.
 12. The networkinterface controller circuitry of claim 11, wherein: the data associatedwith the triggering of the timestamp comprises start of frame delimiterdata.
 13. The network interface controller circuitry of claim 12,wherein: the at least one alignment marker comprises codeword markerdata.
 14. The network interface controller circuitry of claim 13,wherein: the network interface controller circuitry also comprisescircuitry to delete, from the Ethernet protocol frame data, idle controlcharacter data.
 15. The network interface controller circuitry of claim14, wherein: the network interface controller circuitry comprisesphysical medium attachment circuitry and physical coding sublayercircuitry for use in association with processing the Ethernet protocolframe data.
 16. Server computer for use with a physical medium, thephysical medium being for use in Ethernet protocol frame transmissionbetween the server computer and a remote device, the server computercomprising: a system-on-chip comprising: network interface controllercircuitry comprising: circuitry to receive, via the physical medium,Ethernet protocol frame data, the Ethernet protocol frame datacomprising at least one alignment marker and data associated withtriggering of a timestamp; circuitry to detect the data associated withthe triggering of the timestamp; circuitry to trigger, based upon thedata associated with the triggering of the timestamp, the timestamp; andphysical layer circuitry to (1) remove, after the triggering of thetimestamp, the at least one alignment marker from the Ethernet protocolframe data, and (2) to insert, after the triggering of the timestamp, atleast one idle character into the Ethernet protocol frame data.
 17. Theserver computer of claim 16, wherein: the data associated with thetriggering of the timestamp comprises start of frame delimiter data. 18.The server computer of claim 17, wherein: the at least one alignmentmarker comprises codeword marker data.
 19. The server computer of claim18, wherein: the network interface controller circuitry also comprisescircuitry to delete, from the Ethernet protocol frame data, idle controlcharacter data.
 20. The server computer of claim 19, wherein: thenetwork interface controller circuitry comprises physical mediumattachment circuitry and physical coding sublayer circuitry for use inassociation with processing the Ethernet protocol frame data.